Multimodal Musician Recognition

This research is an initial effort in showing how a multimodal approach can improve systems for gaining insight into a musician’s practice and technique. Embedding a variety of sensors inside musical instruments and synchronously recording the sensors’ data along with audio, we gather a database of gestural information from multiple performers, then use machine-learning techniques to recognize which musician is performing. Our multimodal approach (using both audio and sensor data) yields promising performer classification results, which we see as a first step in a larger effort to gain insight into musicians’ practice and technique.